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Risk,  Uncertainty,  and  Trouble 

Escaping  the  RUT  of  Program  Instability 

Col.  Brian  Shimel,  USAF 


When  we  say  we  are  stuck  in  a  rut,  we  imply 
our  often  repeated  actions  are  forced  on  us 
by  things  outside  our  control.  As  if  to  say, 
we  are  not  really  as  crazy  as  we  look.  But 
what  if  we  build  our  own  rut  and  then  act 
as  if  we  have  no  choice? 

When  we  talk  about  things  that  will  happen  in  the  future, 
we  must  remember  those  things  have  not  yet  happened. 
Because  they  have  not  happened,  we  must  allow  for  the 
possibility  they  may  happen  differently  from  the  way  we 
expect,  at  different  times,  or  not  at  all. 

We  cannot  relieve  ourselves  of  the  need  to  plan  for  the 
future  just  because  the  future  is  uncertain.  For  our  plans 
to  be  reasonably  accurate  and  reliable,  it  is  prudent  we 
base  them  on  rational  analysis  and  not  on  wishful  think¬ 
ing.  Unfortunately,  we  do  not  always  think  clearly  about 
the  future.  Our  assumptions  are  often  clouded  by  lapses 
of  judgment,  eternal  optimism  or  dark  pessimism,  and 
trepidation  about  admitting  risk  and  uncertainty  to  those 
around  us.  In  many  instances,  we  shy  away  from  accept¬ 
ing  the  full  impact  of  risk  and  uncertainty  on  future  con¬ 
ditions  because  we  become  overly  concerned  that  our 
ideas  will  be  rejected  unless  we  can  guarantee  successful 
results. 

When  we  talk  about  the  future,  “risk”  is  the  term  used  to 
discuss  a  possible  negative  outcome  of  an  unfavorable 
event  or  action,  while  “uncertainty”  refers  to  the  unknown 
variability  around  a  prediction  of  a  future  state.  Potential 
risk  causes  us  to  set  aside  resource  reserves  to  help  over¬ 
come  possible  setbacks.  Uncertainty  causes  us  to  make 
assumptions  about  what  may  happen  and  estimate  how 
valid  our  assumptions  will  prove  to  be.  Risk  and  uncer¬ 
tainty  are  not  what  get  us  into  trouble.  We  get  into  trouble 
when  we  ignore,  or  unwisely  discount,  risk  and  uncer¬ 
tainty.  I  call  that  Risk,  Uncertainty,  and  Trouble— and  it  is 
a  RUT  of  our  own  making. 

Making  Assumptions 

The  Department  of  Defense  is  upgrading  and  improving 
its  capabilities.  To  fund  this  investment,  the  department 
is  looking  for  efficiency  and  taking  reductions  across  all 
areas  of  operations.  It  is  vitally  important  we  understand 


“However  beautiful  the 
strategy,  you  should 
occasionally  look  at  the 
results . 99  — Winston  Churchill 

how  our  modernization  money  is  being  spent  so  we  can 
get  the  best  possible  results  from  our  investment  and 
minimize  the  impact  to  the  rest  of  the  department. 

Financial  instability  is  a  problem.  Budgets  are  tight,  the 
pace  of  operations  is  high,  and  short-notice  changes  pop 
up  against  a  list  of  requirements.  Leaders  in  acquisition 
are  trying  to  insulate  programs  from  financial  instability. 
To  avoid  overruns,  they  have  asked  for  extra  funding  to 
reach  a  higher  confidence  level  in  costs  and  scheduling. 
Many  acquisition  programs  have  moved  from  funding  at 
the  traditional  50-percent  confidence  level  estimate  to 
an  80-percent  confidence  level  to  provide  better  budget 
stability  and  avoid  costly  program  failures. 

The  biggest  problem  we  have  in  establishing  a  baseline 
and  predicting  the  cost  of  a  weapon  system  to  develop 
and  deliver  it  to  the  warfighter  is  that  we  know  too  little 
about  the  undeveloped  weapon  system  and  the  difficul¬ 
ties  we  are  going  to  face  getting  it  into  the  field.  While  it 
may  be  human  nature  to  worry  excessively  about  things 
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we  do  not  understand  well,  another  common  reaction  is 
to  discount  the  impact  of  uncertain  challenges. 

Even  with  the  well-documented  loss  of  organic  cost  ana¬ 
lysts,  we  still  often  perform  reasonable,  initial  analyses 
of  risk  and  uncertainty  The  RUT  of  program  instability 
is  caused  more  by  our  resistance  to  fully  communicating 
risk  and  uncertainty  to  decision  makers  than  by  our  lack 
of  ability  to  capture  it.  As  information  is  entered  into  our 
approval  and  decision-making  system  and  rises  through 
the  corporate  process,  the  underlying  risk  and  uncertainty 
are  often  watered  down  in  a  series  of  negotiations  forced 
by  severe  budget,  time,  and  resource  competition.  We  fear 
that  risk  or  uncertainty  will  weaken  our  chance  to  gain 
funding  or  approval  for  our  proposed  course  of  action,  and 
we  back  away  from  clearly  expressing  measures  of  risk 
and  uncertainty  that  are  subjective  by  their  very  nature. 

Like  it  or  not,  the  system  drives  us  towards  a  point  esti¬ 
mate,  and  when  a  budget  is  laid  in  against  it,  all  future 
programmatic  success  or  failure  is  measured  against  what 
is  only  a  reasoned  compromise.  Time  and  time  again,  that 
is  what  gets  us  into  trouble. 

Embracing  the  Full  Picture 

One  reason  we  shy  away  from  fully  explaining  risk  and 
uncertainty  is  that  they  are  perceived  as  bad  news.  We 
often  discount  their  very  existence  or  impact.  Failing  to 
embrace  the  true  condition  of  any  situation  leads  to  a 
cycle  of  mistaken  assumptions  and  improper  priorities 
that  can  sabotage  a  manager’s  chance  of  addressing  the 
real  issues,  and  the  chain  of  command’s  chance  to  provide 
meaningful  support  early  enough  to  make  a  difference. 

Assumptions  must  be  correlated  with  evidence  in  order 
to  build  a  realistic  baseline.  If  we  are  to  avoid  the  RUT 
of  program  instability,  we  must  accurately  describe  the 
uncertainty  and  risk  a  program  faces,  and  we  must  ad¬ 


dress  those  items.  That  is  how  we 
will  gain  the  smooth  traction  of 
high-confidence  acquisition  pro¬ 
grams. 

For  example,  if  we  develop  a  plan 
to  solve  a  technical  problem  and 
give  ourselves  a  reasonable  time 
to  accomplish  the  task,  we  tend 
to  discount  the  risk  involved  in 
actually  solving  the  problem.  As 
schedule  risk  and  technical  risk 
are  highly  correlated,  this  ten¬ 
dency  leads  us  to  assume  away  a 
significant  portion  of  the  risk  by 
planning  a  development  timetable 
that  appears  reasonable  to  us  but, 
in  many  cases,  is  not  when  seen 
in  the  context  of  what  must  be 
accomplished  and  in  the  context 
of  real-world  capabilities.  So,  we  end  up  underestimating 
that  portion  of  the  risk.  Next,  we  estimate  the  80  percent 
confidence  interval  of  a  fraction  of  the  risk — the  risks 
associated  only  with  the  estimating  equations— and  de¬ 
clare  we  have  80  percent  confidence  in  the  development 


We  get  into  trouble  when 
we  ignore,  or  unwisely 
discount,  risk  and 
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estimate.  When  we  minimize  the  risk  and  uncertainty 
of  our  program  to  the  approving  officials  in  our  chain  of 
command,  they  make  biased  decisions  based  on  “opti¬ 
mistic  assumptions”  (our  current  euphemism  for  poor 
judgment). 

Getting  into  Trouble 

Because  of  the  amount  of  risk  and  uncertainty  inherent  in 
a  weapons  system  development  program,  the  amount  of 
extra  money  needed  to  go  from  a  50  percent  confidence 
that  the  program  will  not  exceed  a  certain  cost  to  a  higher 
confidence  level  is  often  unaffordable.  For  example,  look 
at  the  Actual  Space  Program  Cost  Distribution  figure  on 
the  previous  page,  which  illustrates  three  different  esti¬ 
mates  made  of  its  total  cost. 

The  scale  of  the  figure  is  millions  of  fiscal  year  2002  base 
year  (uninflated)  dollars.  The  contractor  bid  to  deliver  this 
program  at  a  stated  50  percent  confidence  that  the  cost 
would  be  $6  billion  or  less.  Given  the  narrow  range  of 
uncertainty  assumed  by  the  contractor  (the  yellow  line), 
it  would  take  only  another  3  percent  of  funding  to  gain 
80  percent  confidence  that  the  program  would  finish  at 
or  under  $6.2  billion.  The  program  office  did  its  own 
estimate  and  predicted  that  the  cost  of  delivery  would 
be  approximately  $6.4  billion  at  50  percent  confidence 
(the  blue  line).  Given  the  program  office’s  assumed  un¬ 
certainty,  it  would  take  only  an  extra  6  percent  of  fund¬ 
ing  ($400  million)  to  reach  80  percent  confidence.  The 
program  then  went  through  an  independent  review.  The 
independent  cost  estimate  predicted  a  cost  of  $7.7  bil¬ 
lion  with  another  10  percent,  or  $8.5  billion,  to  reach  80 
percent  confidence  (the  red  line). 


Many  would  look  at  the  range  of  costs  developed  for  the 
independent  review  as  being  surprising  when  compared 
to  the  earlier  estimates.  It  is  easy  to  imagine  the  program 
office  and  contractor  complaining  bitterly  that  the  inde¬ 
pendent  estimators  did  not  fully  understand  their  cost¬ 
saving  initiatives  and  managerial  skill. 

And  it  turns  out,  the  independent  estimators  were  wrong: 
Even  their  distribution  was  far  too  narrow.  In  fact,  when 
predicting  the  price  of  a  commodity  as  simple  as  a  carton 
of  eggs  five  years  into  the  future,  there  is  a  standard  error 
of  1 5  percent. 

Because  one  standard  error  represents  roughly  the  dif¬ 
ference  between  50  percent  and  80  percent  confidence, 
in  order  to  be  80  percent  confident  that  you  will  have 
enough  money  to  pay  for  a  dozen  eggs  five  years  from 
now,  you  will  need  to  hold  1 5  percent  more  than  the 
expected  price.  Now  imagine  how  much  larger  the  stan¬ 
dard  error  is  for  our  sophisticated,  state-of-the-art  weapon 
systems  that  will  take  more  than  a  decade  to  develop  and 
procure.  Because  the  example  in  our  graph  is  based  on 
real  numbers,  you  might  be  curious  to  know  that  the  latest 
cost  estimate  for  the  program  exceeded  $  1 3  billion  before 
it  was  de-scoped.  But  it’s  not  all  about  the  cost  estimatel 

The  cost  growth  in  this  program  was  the  result  of  optimis¬ 
tic  assumptions  associated  with  technology  levels,  integra¬ 
tion  complexity,  and  cost.  Risk  and  uncertainty  were  un¬ 
derestimated  and  the  program  was  funded  at  something 
less  than  even  the  50-percent  confidence  level. 

It  does  not  mean  the  cost  estimators  should  have  esti¬ 
mated  an  82  percent  increase  in  costs  to  go  from  50 
percent  confidence  to  80  percent.  Successfully  bounding 
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the  upper  limits  of  a  program  estimate 
requires  programmatic  control,  not 
throwing  good  money  after  bad.  The 
summary  purpose  of  that  example  is 
to  show  that  nobody  fully  understood 
or  communicated  the  cost  and  technical 
risk  associated  with  the  program.  Deci¬ 
sion  makers  were  hamstrung  by  poor 
information  and  a  culture  of  optimistic 
assumptions.  We  got  into  this  trouble  by 
minimizing  the  risk  and  uncertainty  of 
new  technology  that  was  being  devel¬ 
oped  for  this  system. 

Keeping  an  Eye  on  the 
Customer 

Sometimes  we  have  good  reasons  for 
rushing  things  to  the  field.  An  extremely 
important  part  of  this  discussion  must 
be  that  new  programs  are  often  built 
from  urgent  warfighter  needs.  There  is  no  way  anyone 
in  the  business  of  DoD  weapon  systems  development 
wants  to  let  our  troops  in  theater  down!  Both  in  the  heat 
of  the  battle  and  in  the  heat  of  getting  better  capabilities 
to  those  in  the  battle,  cost  and  schedule  risks  are  all  too 
often  understated  under  the  pressure  of  mission  accom¬ 
plishment.  It  is  a  disservice  to  our  leadership  to  think  they 
won’t  accept  the  risks  if  we  communicate  them  and  let 
them  debate  whether  the  potential  benefits  are  worth 
it— or  not. 

Accept  and  Control,  Not  Escape 

Risk  and  uncertainty  are  perceived  as  bad  news.  We  fight 
risk  and  uncertainty  tooth  and  nail.  It  would  be  wiser  to 
consider  risk  and  uncertainty  as  a  giant  rubber  band— the 
more  you  pull  away  from  them,  the  harder  they  pull  back 
on  you.  Failing  to  admit  that  things  may  not  proceed  ex¬ 
actly  according  to  plan  is  a  recipe  for  trouble.  Many  things 
in  our  business  are  unknown  and  will  stay  unknown  until 
we  attempt  to  execute  a  program.  After  all,  don’t  we  try 
to  put  state-of-the-art  technology  into  new  weapon  sys¬ 
tems? 

We  are  developing  risky  technology  on  aggressive  sched¬ 
ules  and  claiming  stable  management  environments.  It 
just  doesn’t  make  sense.  Adding  money  to  the  top  line  of 
an  effort  that  is  not  fully  understood  is  prohibitively  ex¬ 
pensive.  Optimistic  assumptions  must  be  correlated  with 
evidence  in  order  to  build  a  realistic  baseline. 


The  decisions  that  make  a  system  affordable  must  be 
based  on  more  than  wishful  thinking.  We  should  be 
straightforward  about  the  risk  we  are  proposing  to  take 
on.  Strongly  and  clearly  communicating  risk  and  uncer¬ 
tainty  up  the  chain  of  command  will  help  leadership  make 
better  strategic  decisions.  That  will  result  in  improved  use 
of  resources  and  greater  combat  effectiveness.  If  we  are 
to  get  out  of  our  RUT,  the  first  step  will  be  to  recognize 
we  are  in  one.  We  should  be  motivated  to  take  the  risk  of 
communicating  better  and  more  balanced  information 
to  decision  makers.  Not  every  idea  is  worth  a  full-scale 
development  effort,  and  there  is  nothing  wrong  with  ad¬ 
mitting  that.  We  must  think  clearly  about  uncertainty  and 
risk,  and  we  must  fight  the  temptation  to  discount  those 
factors  when  communicating  the  real  conditions  of  our 
management  situation.  We  don’t  get  in  trouble  because 
of  risk  and  uncertainty.  We  get  in  trouble  for  not  admit¬ 
ting  to  ourselves— and  those  who  rely  on  us— all  of  the 
risk  and  uncertainty  that  inherently  exist  in  everything 
we  plan  to  do. 

Note:  Between  December  2007  and  August  2008,  the 
price  of  eggs  increased  by  more  than  30  percentl 

The  author  is  especially  indebted  to  assistance  from  Jay  Jor¬ 
dan, ,  technical  director  of  the  Air  Force  Cost  Analysis  Agency 
The  graph  is  from  Jordan’s  excellent  briefing,  “Cost  Estimate 
Quality  and  Confidence.  ” 


It  is  a  disservice  to  present  a  decision  maker  with  an 
estimate  for  a  new  groundbreaking  weapon  system  that 
claims  the  system  can  be  developed  for  a  certain  price 
and  that  the  confidence  can  go  from  50  percent  to  80 
percent  confidence  with  only  a  3  percent  or  a  6  percent 
increase  in  funding.  And  it  is  foolhardy  for  a  decision 
maker  to  accept  that  estimate. 


The  author  and  Jay  Jordan  welcome  comments 
and  questions  and  can  be  contacted  at  brian. 
shimel@hanscom.af.mil  and  jay.jordan@ 
pentagon.af.mil.  (Please  send  the  hard  questions 
to  Mr.  Jordan!) 
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